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A  Response  Model  for 
Multiple  Choice  Items 

In  practical  ability  measurement,  binary  item  response  models  have 
been  applied  routinely  to  data  from  multiple-choice  tests  with  four  or 
five  alternatives  per  item.  To  use  the  binary  models,  the  data  have 
been  dichotomized  (correct  and  incorrect)  and  the  distinct  identity  of 
the  incorrect  alternatives  has  been  lost.  This  procedure  logically  fol¬ 
lows  the  tradition  of  "scoring"  a  test  by  considering  only  the  correct 
answers;  however,  the  binary  models  are  incomplete  as  theories  of  the 
item  responses  as  useful  information  may  be  lost  in  the  dichotomization. 

It  is  possible  to  conceive  of  a  unidimensional  latent  variable 
model  which  completely  explains  the  item  response  data,  and  several  have 
been  proposed.  The  model  which  will  be  introduced  here  is  an  extension 
of  that  proposed  by  Bock  (1972)  and  subsequently  extended  by  Samejima 
(1979);  a  different  parameterization  of  the  same  model  has  been 
discussed  by  Sympson  (1983).  Bock  (1972),  Thissen  (1976),  Sympson 
(1983),  and  Levine  and  Drasgow  (1983)  have  shown  that  some  increased 
precision  of  measurement  may  be  obtained  when  information  in  the  incor¬ 
rect  alternatives  on  multiple  choice  tests  is  included  in  the  item 
response  model.  In  a  subsequent  section,  we  will  discuss  this  gain  in 
information  with  a  new  model,  as  well  as  some  attendant  difficulties. 

In  the  next  section  we  will  introduce  the  model  and  describe  a  workable 
scheme  for  maximum  likelihood  estimation  of  its  parameters. 
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The  Model 


The  model  for  multiple-choice,  items  to  be  described  makes  heavy 
use  of  the  multivariate  logistic  transformation  suggested  as  an  item 
response  model  by  Bock  (1972),  so  it  is  useful  to  begin  with  a  clear  un¬ 
derstanding  of  that  model.  For  m  categorical  responses,  we  specify  m 
response  functions  z^  =  a^Q  +  c^,  each  of  which  is  a  linear  function  of 
the  latent  ability  variable  0.  Usually  2correct  would  have  a  positive 
slope  and  the  other  z^'s  would  have  lesser  positive  or  negative  slopes. 
The  linear  functions  describe  one  of  the  simplest  possible  relationships 
between  the  response  and  8.  To  make  this  a  model  for  categorical  item 
responses,  the  zfc  (which  lie  on  the  real  line)  must  be  mapped  into 
[0,1].  This  is  accomplished  by  the  multivariate  logistic  transforma¬ 
tion,  so 


P(x.=h|0;a,c) 

J 


exp(zh) 


I  exp(z,  ) 

k=l,m.  K 
J 


(1) 


The  function  (1)  is  Bock's  (1972)  model  for  an  item  response,  x.=h,  in 

J 

which  h  =  1,2,... m.  for  a  multiple  choice  item  j  with  m.  (classes  of) 
response  alternatives.  The  item  parameters  are  the  vectors  a  and  c, 
subject  to  two  suitable  linear  constraints  (see  below),  giving  2(nij-l) 
free  parameters.  The  model  described  by  (1)  is  moderately  flexible,  but 
lacks  flexibility  in  certain  crucial  respects  at  the  extremes. 
Specifically,  one  of  the  response  alternatives  must  have  the  largest 
positive  value  of  a,;  the  trace  line  for  that  alternative  is  then 

fv 

monotonic  increasing.  That  may  be  theoretically  acceptable,  since  it 
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is  probably  the  correct  response.  However,  another  alternative  must 


have  the  largest  negative  value  of  a^  (la^=0  is  one  of  the  linear  con¬ 


straints),  and  that  alternative's  trace  line  must  be  monotonic 


decreasing.  The  latter  aspect  of  the  model  is  less  acceptable:  it  im¬ 


plies  that  as  ability  decreases  the  probability  of  selecting  one  par¬ 


ticular  incorrect  response  approaches  unity,  and  all  the  others  go  to 


zero.  That  is  unlikely. 


Samejima  (1979)  proposed  a  solution  to  this  problem  in  a  concep¬ 


tual  modification  of  Bock's  (1972)  model  in  which  she  added  a  completely 


latent  response 


$ory  labelled  "zero."  Here,  we  will  sometimes  refer 


to  this  category  as  "don't  know"  (DK) .  Lord  (1982)  has  introduced  a 


similar  conceptual  entity  which  he  describes  by  saying  the  examinee  is 


"(totally)  undecided."  In  Samejima 's  model,  DK  is  not  an  observed 


response,  but  a  latent  one,  multi-logit-linearly  dependent  on  a  latent 


trait  (giving  two  layers  latent) ;  the  idea  was  that  some  proportion  d. 


(=l/nij  in  Samejima  (1979))  "guessed"  each  of  the  observable  response  al¬ 


ternatives  and  were  "mixed  in"  with  the  examinees  who  chose  those  alter¬ 


natives  intentionally.  So  the  model  becomes 


P(Xj=h|8;a,c,d)  = 


exp(zfa) 


dh  exp(V 


I  exp(z,  ) 

k=0,ra.  K 
J 


I  exp(z.  ) 

k=0,m.  K 
J 


in  which  the  second  term  adds  a  proportion  d^  of  those  who  "don't  know" 


into  each  trace  line;  call  it  Pj(h)  for  future  brevity: 


exp(z.)  +  d,  exp(zn) 
P.  (h)  =  h  h  0 

J  I  exp(z,  ) 

k=0 , ra . 

J 
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in  which  h  takes  the  values  1,2,...hk.  Samejima's  (1979)  model  had  two 


more  parameters  than  Bock's  (1972)  model:  and  cQ.  In  her  presenta¬ 


tion,  the  d^  were  fixed  and  equalled  l/m^ ;  this  represents  the 

hypothesis  that  those  of  sufficiently  low  ability  assign  their  responses 

randomly  with  equal  probability  to  each  of  the  response  alternatives. 

We  found  that  unlikely;  later  we  will  show  that  it  is  not  empirically 

the  case.  So  we  extend  the  version  of  the  model  given  in  (2)  to  allow 

the  d,  ,  h=l,2,...m.  to  be  functions  of  estimated  parameters. 

1 


Indeterminacies  and  Constraints 


The  model  expressed  in  (2)  has  a  number  of  indeterminacies  and  re¬ 
quires  the  imposition  of  some  constraints  to  become  identifiable.  The 
linear  constraints  required  on  a  and  c, 


2  a  =  Ic.  =  0 
k=0,m.  k=0,m. 

J  J 


are  imposed  by  reparameterization: 


a  =  To 


and 


£  =  TI 


where  T  is  an  (m^+l)  x  ok  transformation  matrix,  and  a  and  j  are  vec¬ 


tors  of  free  parameters  of  order  ;  T  is  the  transpose  of  that  in  Bock 


(1972)  throughout.  There  is  a  further  indeterminacy  with  respect  to  the 
sign  of  a;  this  is  an  indeterminacy  of  reflection  of  the  latent  variable 
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9  which  is  logically  identical  to  the  rotational  indeterminacy  of  common 
factor  analysis.  The  reflection  indeterminacy  is  not  explicity  solved, 
but  rather  the  iterative  solution  of  the  likelihood  equations  is  started 
with  a^  positive  for  h  =  the  correct  response;  this  is  usually  suf¬ 
ficient  to  keep  the  estimated  solution  "right-side-up." 


be  constrained  to  sum  to  unity  and  lie  on  the  interval  [0,1].  The  dual 
constraint  is  imposed  in  two  parts.  In  the  first,  the  same  multivariate 
logistic  used  in  the  model  is  also  employed  to  make  the  d^  proportions 
from  a  set  of  psuedo-parameters  d^*,  which  lie  on  the  real  line: 

dfa  =  exp(d  *)  /  I  exp(d  *)  .  (3) 

k-1>*j  ‘ 

The  parameters  are  on  the  real  line,  but  like  a  and  c,  they  must 
satisfy  Xd^*  =0.  So 

d*  =  T25  (4) 


in  which  T~  is  an  m.  x  (m.-l)  transformation  matrix  of  the  same  form  as 
2  J  J 

T  (above)  but  of  lesser  order,  and  6  is  a  vector  of  length  - 1 )  of 
parameters  which  (through  (4)  and  (3))  give  the  proportions  d^. 

So  the  set  of  free  parameters  for  an  item  consists  of  m^ 
("a-contrasts") ,  m.  y.’s  ("c-contrasts") ,  and  (m.-l)  5. 's 

J  K  J  K 

("d-contrasts") ,  for  a  total  of  3nij-l  parameters:  11  for  a  four- 
alternative  multiple-choice  item. 


The  estimation  system  we  use  permits  any  of  the  elements  of 
{a , ^ , 6 }  to  be  fixed  at  any  constant  value,  or  to  be  constrained  to  be 
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equal  to  any  other  parameter.  With  this  facility,  our  model  (2)  in¬ 
cludes  the  previous  models  hierarchically  as  subsets.  If  5  is  fixed  at 
0,  than  all  the  d^l/oK  and  (2)  becomes  Samejima's  (1979)  model.  If  the 
parameters  ag  and  Cg  are  fixed  at  zero,  the  d^  become  irrelevant  (and 
are  deleted  mechanically)  and  the  model  becomes  Bock's  (1972)  model.  In 
our  examples,  we  will  compare  the  goodness  of  fit  of  all  of  these 
models,  as  well  as  versions  of  (2)  that  include  equality  constraints  on 
particular  parameters  imposed  across  items. 

The  Relationship  of  the  Model  to  Data 

Models  in  item  response  theory  (IRT)  are  intended  to  explain  ob¬ 
served  covariation  among  test-item  responses.  Unidimensional  "latent 
trait"  theories  explain  that  covariation  by  appeal  to  an  underlying  la¬ 
tent  variable  (usually  denoted  8)  on  which  the  probabilities  of  the 
responses  are  functionally  dependent.  Each  pair  of  item  responses  is 
theorized  to  have  non-zero  covariance  because  the  probability  of  both 
item  responses  depends  on  0.  The  models  usually  hold  that  0  is  the  sole 
cause  of  covariation  among  the  item  responses;  conditional  on  8,  the 
item  responses  are  theorized  to  be  independent  ("local  independence"). 

A  complete  IRT  model  for  the  item  response  data  for  a  multiple- 

choice  test  in  which  n  items  each  have  m  alternatives  would  be  a  model 

for  the  covariation  in  the  mn  contingency  table  containing  the  counts 

2 

of  respondents  giving  each  response  pattern.  Such  a  4  table  for  two 
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four-alternative  multiple-choice  items  is  given  in  Table  1. 


Insert  Table  1  About  Here 


Evidence  that  there  is  some  information  in  the  examinees'  i  rrect 
choices  is  apparent  (or  at  least  can  be  found)  in  the  covai  ice  in  the 

table.  (The  column  percentages  for  the  counts  are  given  i  ?,-entheses 

to  aid  in  interpretation.)  Alternative  C  is  correct  for  item  a;  respon¬ 
dents  who  select  C  on  item  2  are  most  likely  to  choose  B  on  item  1;  B 
is  correct,  so  that  is  double  evidence  of  their  high  ability.  But  ex¬ 
aminees  who  choose  A  or  B  on  item  2  are  more  likely  to  respond  correctly 
to  item  1  (36%  and  34%)  than  those  who  pick  D  on  item  2  (only  25%) .  A 
possible  explanation  is  that  those  who  select  A  or  B  on  item  2  are 
higher  on  the  ability  continuum  than  are  those  who  pick  D.  Even  with 
16  cells,  this  sort  of  argument  may  be  lengthy,  but  the  idea  is  that 
wrong  responses  are  related  to  each  other;  if  the  responses  are  locally 
independent,  this  implies  a  relationship  to  "the  trait." 

The  goodness  of  fit  of  a  multiple  category  IRT  model  to  the  data 

in  such  a  table  may  be  evaluated  with  conventional  likelihood  ratio  x2 

statistics.  For  sets  of  data  with  more  examinees  than  cells  in  the 

table,  the  likelihood  ratio  test  against  a  general  multinomial  alterna- 

35 

tive  may  be  used.  For  larger  tables  (35  four-choice  items  gives  a  4 
table),  there  is  no  "general  multinomial  alternative";  but  the 
likelihood  ratio  test  between  hierarchically  nested  models  still  may  be 
used  to  evaluate  the  significance  of  the  additional  parameters  of  the 
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larger  model. 

Estimation 

In  general,  the  data  for  the  estimation  of  the  parameters  {a, ^,5} . 
for  each  item  j,  j=l,...,n  consist  of  the  counts  of  response  patterns 
rx  (where  each  Xj.may  take  values  in  [1,2,. .  .in.])  in  an  x  m2  x...x 
mQ  contingency  table  like  Table  1.  The  probability  of  observing  a  par¬ 
ticular  response  pattern  x  when  drawing  an  examinee  from  a  population 
in  which  9  follows  a  distribution  t})(9) ,  assumed  N (0 , 1 )  in  the  examples, 
is 

os 

P(x)  =  ;  n  P.(x.)  <K8)  d8  (5) 

-«  j=l,n  J  J 

in  which  P.(x.)  is  from  equation  (2). 

J  J 

The  likelihood  for  the  entire  set  of  observed  data  is 

I  =  C  n  P(x)rx  (6) 

x 

in  which  C  is  a  normalizing  constant  not  dependent  on  the  parameters, 
and  the  product  runs  over  all  possible  response  patterns.  In  practice, 
of  course,  for  more  than  a  handful  of  items  the  number  of  possible 
response  patterns  is  astronomical,  and  the  "count"  in  each  cell  is  one 
or  zero.  In  such  cases,  only  those  response  patterns  actually  observed 
need  to  be  considered  in  the  computation  of  (6),  or  (7),  a  function 
proportional  to  the  loglikelihood: 
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(7) 

For  a  very  small  number  of  items  (3  or  4;  the  parameters  of  the 
full  model  are  not  identifiable  with  only  2  items,  as  there  are  fewer 
cells  than  parameters),  it  may  be  possible  to  obtain  the  MLEs  by 
directly  maximizing  (7)  with  a  Newton-type  algorithm.  Bock  (1972) 
reported  such  results  for  his  original  model  with  procedures  similar  to 
those  used  by  Bock  and  Lieberman  (1970). 

Bock  and  Aitkin  (1981)  described  an  extremely  simple  and  elegant 
algorithm  for  maximizing  functions  like  (7)  for  binary  models,  and  we 
extend  that  algorithm  to  the  estimation  of  multiple  category  models. 

The  Bock-Aitkin  algorithm  is  a  two-step  procedure  like  the  "EM- 
algorithm"  (Dempster,  Laird,  and  Rubin,  1977),  so  we  describe  first  the 
"E-step"  and  then  the  "M-step."  The  procedure  is  iterative,  and  EH-pairs 
are  repeated  until  the  process  converges. 

The  E-step 

The  Bock-Aitkin  algorithm  is  based  on  a  discrete  representation 
of  <j>(9)  and  the  integrand  of  P(x),  both  continuous  densities,  over  Q 
"quadrature  points"  9^,  with  q=l,2,...Q.  Such  a  discrete  representation 
of  the  continuous  densities  may  be  made  arbitrarily  close  to  continuous 
reality  by  choosing  Q  large,  just  as  numerical  integration  may  be  made 
arbitrarily  accurate  by  using  sufficient  quadrature  points.  However, 
large  values  of  Q  slow  the  computations;  Q=10  over  the  range  8=-4.5  to 
9=4.5  in  unit  steps  seems  to  be  sufficiently  accurate  and  fast  for  many 
applications  and  is  used  in  all  of  the  examples  here. 


ZQ  ~  l  z*  logP(x)  . 
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Under  the  assumption  that  the  population  is  composed  of  in¬ 
dividuals  who  are  members  of  Q  discrete  "classes"  with  values  0^,  9^, 

. ..,  0q  on  the  latent  variable,  "complete  data  sufficient  statistics" 
for  the  estimation  of  the  item  parameters  for  item  j  would  con¬ 

sist  of  a  table  of  counts  R  .*,  in  which  each  element  r*.  is  the  number 

2  Jkq 

of  individuals  in  class  0^  selecting  response  alternative  k  on  item  j . 

So  the  E-step  of  the  Bock-Aitkin  algorithm  consists  of  computing  the  ex¬ 
pected  values  of  the  rj^q,  conditioned  on  the  data  and  the  current 
provisional  estimates  of  the  item  parameters. 

Using  provisional  estimates  of  the  item  parameters  for  each  item 
(as  starting  values  we  use  6=0,  J=0,  and  a^-l  for  categories  k  which  are 
incorrect  and  a.  =2  for  category  k  correct) ,  compute  the  elements  of  the 

m.  x  Q  table  R.*  containing: 

J  J 

E^rtko  ldata;{i>i»£})  =  I  r  [P(x;0  )/Z  P(x;0  )]  (8) 

J  4  (x  in  |)  -  4  q  4 


in  which  4  is  the  set  of  x  in  which  x.=k  and 

2 

P(x;8J  =  n  P.(x.)<D(0J  . 


q 


j=l,n 


Note  that,  while  (9)  and  (8)  are  computed  in  a  (potentially  long)  loop 
over  the  observed  response  patterns,  the  values  ( k)  for  each  item  from 
(2)  are  required  only  for  a  fixed  set  of  Q  values  of  0.  If  those  values 
are  placed  in  a  table  before  the  E-step  is  begun,  the  computations  in¬ 
volved  in  (8)  and  (9)  are  limited  to  table  look-up,  multiplication,  and 
addition.  The  E-step  yields  a  set  of  n  ok  x  q  tables  of  non-integral 
artificial  "counts"  which  are  used  as  data  in  the  M-step. 
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The  M-step 

The  M-step  consists  of  maximum  likelihood  estimation  of  the 
parameters  for  all  items  j=l,2,...n,  using  the  tables  of  ex¬ 

pected  values  Rj*  as  data.  It  is  simply  nonlinear  regression. 

In  terms  of  the  "data"  in  R^*,  the  loglikelihood  for  item  j  is 

Vj^lkq  l08VkiV  (10) 

in  which  P.(k;0  )  is  equation  (2)  evaluated  at  0  .  Standard  gradient 

J  q  Q 

methods  may  be  used  to  maximize  iL  over  the  parameter  space.  The  MIEs 
of  the  parameters  are  obtained  where 
32. 

J  =  0 

ar 


for  all  parameters  in 

strained  by  T  such  that  k=T£, 

32.  32. 

J  =  j  J 

W  ai r 


Since  for  any  set  of  parameters  con- 


(11) 


we  will  state  the  required  components  of  the  gradient  for  the  con¬ 
strained  parameters  {a,c,d};  the  gradients  used  to  maximize  (10)  are 
then  given  by  (11).  Further, 


32. 

J 

§1T 


1  3Vk;V 

P . (k;0  )  3k 

j  q 


(12) 


So  what  we  really  need  are  the  derivatives  of  (2)  with  respect  to  a,  c, 
and  d*  (substitute  all  for  K  above).  To  write  these,  a  useful  bit  of 
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shorthand  is 

ek  =  exp(ak0  +  ck)  , 

which  is  always  assumed  to  be  evaluated  at  the  appropriate  value  of  9; 
all  summations  of  ek  are  over  categories  k=0,l,...m^.  The  elements  of 
the  gradient  (P^  is  short  for  Pj(h;9^)  hereafter)  to  be  substituted  in 
(12),  then  (11),  and  finally  zeroed  to  maximize  (10)  are: 
dK  [(2eJ8d ue„  -  (eu  +  due„)8ert] 


[(ZejSe.  -  (e.  ♦  d.eJ8e.  ] 


[(Xejdven  -  (eK  +  dve„)ert] 
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3Ph  ej  Xdk*  -  dh*2 

_nr ' 

The  gradient  does  not  simplify  very  much  since  the  model  is  not 

part  of  an  exponential  family.  However,  it  is  possible  to  locate  the 

inavi mum  of  2.  using  these  derivatives,  and  use  the  resulting  set  of 
J 

parameters  in  the  next  E-step.  Our  estimation  procedure  allows  equality 
constraints  to  be  placed  across  as  well  as  within  items;  this  requires 
some  of  the  gradients  to  be  accumulated  across  items,  but  causes  no 
serious  problem  beyond  book-keeping.  We  use  a  conditioned  Newton-type 
algorithm  (MINIM,  described  by  Haberman,  1974)  to  locate  the  maximum  of 
(10);  the  conditioning  is  useful  since  the  matrix  of  second  derivatives 
may  be  nearly  singular  for  some  items. 

Convergence  and  Local  Minima 

Using  the  updated  item  parameters  from  the  M-step,  the  sequence 
(E-step,  M-step]  is  repeated  until  either  a)  the  parameters  stabilize 
or  b)  a  fixed  number  of  cycles  is  reached  (we  usually  use  15  or  20). 

The  algorithm  sometimes  converges  in  the  parameter  space.  More  often, 
it  does  not,  but  the  lack  of  convergence  causes  no  real  problems.  In 
such  cases,  most  of  the  parameters  (and  the  loglikelihood)  remain 
roughly  constant  after  ten  or  fifteen  cycles,  while  a  parameter  (or  a 
few)  change  linearly  and  indefinitely.  The  changing  parameter  is  fre¬ 
quently  a  particular  or ^  (and  its  associated  y^)  >  changing  as  the  as¬ 
sociated  a^  rises  toward  a  very  high  (possibly  infinite)  MIE.  The  dif¬ 
ference  in  goodness  of  fit  with  such  a  slope  high  (3.  to  5.  or  so)  and 
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much  higher  appears  to  be  negligible,  so  there  is  no  loss  in  simply 
stopping  the  estimation  procedure  arbitrarily.  In  such  a  case,  the 
trace  line  is  fairly  well-determined;  but  the  numerical  value  of  the  as¬ 
sociated  a^  is  not. 

A  second  case  in  which  the  parameters  fail  to  clearly  converge 
arises  with  associated  with  a^'s  near  zero.  The  parameter  c^ 
becomes  ill-defined  when  a^  is  zero,  and  the  associated  y^  "wanders." 
Again,  there  is  no  loss  of  fit  if  the  estimation  procedure  is  stopped. 
The  Bock-Aitkin  algorithm  may  be  speeded  slightly  using  acceleration  as 
in  Thissen  (1982),  but  the  value  of  the  acceleration  parameter  should 
be  limited  to  about  -1,  which  doubles  the  step-size  at  each  cycle,  to 
avoid  oscillation. 

The  likelihood  surface  for  the  model  clearly  has  more  then  one 
"local"  minimum.  The  indeterminacy  with  respect  to  reflection  of  0 
gives  two  equal,  identical  modes.  With  poorly  chosen  starting  values, 
the  estimation  procedure  has  located  other  (apparent)  stationary  points; 
usually  they  seem  to  be  located  in  a  peculiar  region  corresponding  to 
some  items  using  both  reflections  of  6  in  the  same  solution  —  these  may 
be  modes  or  saddle  points.  Good  starting  values  and  the  EM-like  nature 
of  the  Bock-Aitkin  algorithm  provide  a  solution  in  this  case,  because 
EM-algorithms  only  climb  local  modes.  If  the  algorithm  starts  near  the 
desired  part  of  the  likelihood  surface,  it  will  end  there.  It  appears 
that  sensible  starting  values  solve  the  prohlem. 

The  likelihood  surface  is  multi-modal  and  may  be  ill-behaved, 
producing  parameter  estimates  that  are  sometimes  undefined  or  are  oa  the 
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boundary.  The  entire  system  begs  for  full  Bayesian  treatment,  with  a 
prior  distribution  restricting  the  parameters  to  a  reasonable  part  of 
the  space  under  all  circumstances.  At  this  stage  however,  it  is  not 
clear  what  sort  of  prior  may  be  appropriate.  After  some  experience  with 
the  model  is  gained  beyond  that  in  the  examples  which  follow,  we  may  be 
in  a  position  to  specify  reasonable  parameters  for  a  prior. 

Alternatively,  a  different  model,  in  the  exponential  family  and 
with  a  unimodal  likelihood  (preferably  with  similar  properties  to  this 
one  in  terms  of  fitting  the  data),  would  also  solve  all  of  these 
problems.  Such  a  model  has  yet  to  be  proposed. 

Characterizing  9 

The  parameter  estimation  described  above  usually  has  the  goal  of 
"calibrating"  a  set  of  test  items,  after  which  the  parameters  are  to  be 
taken  as  known  and  used  to  characterize  0  for  examinees  who  produce  a 
particular  response  pattern  x.  Given  a  set  of  item  parameters,  the 
posterior  density  for  9  is 

P(x;0)  =  n  P.(x.)<D(0)  ,  (13) 

j*l,n  J  J 

in  which  Pj(xj)  from  equation  (2).  If  the  model  is  correct,  (13) 
describes  the  distribution  of  examinees  who  respond  with  pattern  x.  It 
can  be  characterized  graphically,  and  two  examples  will  be  presented 
below  in  Figure  5. 

For  more  than  a  few  items,  (13)  is  roughly  Gaussian  in  shape,  and 
so  it  may  also  be  described  by  estimates  of  its  location  and  spread.  An 
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extension  of  the  procedure  commonly  used  in  binary  IRT  is  to  use  the 

mode  as  an  estimate  of  the  location  of  (13),  where 

31ogP(x;0)  31ogP(x.;6)  31og<K9) 

=  I  J  + 

56  j=l ,n  50  36 

is  zero,  with  the  variance  approximated  by  the  negative  inverse  of 

32logP(x;9) 

E 

The  modal  estimate  is  practical  to  compute  as  long  as  d>(6)  is  a 
reasonable  function,  and  easy  if  <j>(0)  is  normal.  There  is  no  guarantee 
that  (13)  is  unimodal;  for  small  numbers  of  items  it  is  likely  to  have 
more  than  one  mode.  Multimodality  presents  potential  problems  for 
mechanical  use  of  modal  estimates. 

However,  it  is  also  fairly  straightforward  to  numerically  in¬ 
tegrate  (13)  to  obtain  its  mean  and  variance.  The  mean  has  been  called 
the  "EAP"  (Expected  A  Posteriori)  estimate  of  0  by  Bock  and  Mislevy 
(1982).  An  advantage  of  the  EAP  procedure  over  modal  estimation  is  that 
the  derivative  of  $(9)  is  not  required;  therefore  <t(0)  may  take  any  form 
describable  as  a  histogram  with  finite  variance. 

Examples 

To  illustrate  application  of  the  model  to  item  response  data,  we 
have  analyzed  several  subsets  of  items  from  the  ASVAB  Form  8A  ad¬ 
ministered  to  a  national  probability  sample  of  youth  by  the  National 
Opinion  Research  Center;  the  data  are  described  by  Bock  and  Mislevy 

t 

f 

I 
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(1981).  The  data  used  here  are  from  a  10%  subsample  of  the  NORC  sample, 
total  N=1178.  For  small  illustrations  of  the  performance  of  the  model, 
we  use  four-item  subsets  consisting  of  the  last  four  items  of  the  Word 
Knowledge  (WK)  and  General  Science  (GS)  sections  of  the  test.  Four 
items  were  used  because,  with  roughly  a  thousand  examinees,  that  is  the 
largest  number  for  which  the  number  of  cells  in  the  ma  table  is  less 

4 

than  N  (4  =256),  so  significance  tests  of  the  overall  goodness  of  fit 
of  the  model  against  a  general  multinomial  alternative  are  feasible. 

The  last  items  were  used  because  they  are  most  difficult,  giving  the 
trace  lines  more  latitude  to  be  fitted. 

For  the  four-item  examples,  only  examinees  with  complete  data  (no 
non-response)  for  all  four  items  were  used.  For  GS,  N=1048  and  214  of 
the  256  cells  of  the  table  were  filled  (one  of  the  4x4  marginal  tables 
of  this  table  makes  up  Table  1);  for  WK,  N=976  and  only  156  response 
patterns  were  observed. 

We  also  fitted  several  forms  of  the  model  to  the  entire  set  of  35 
WK  items,  and  a  subset  of  12  of  the  25  GS  items  called  "physical 
science"  hereafter.  (The  ASVAB  "General  Science"  section  consists  of 
about  half  biological  and  half  physical  science  questions;  we  selected 
a  subset  more  likely  to  be  unidimensional.)  In  these  analyses,  all  ex¬ 
aminees  were  used  (N=1178)  and  non-response  data  to  individual  items 
were  ignored  (not  placed  in  any  category) . 


! 


! 
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Results- 

Table  2  gives  summary  goodness  of  fit  results  for  the  four-item 
examples . 


Insert  Table  2  About  Here 


Bock's  (1972)  model  and  Samejima's  (1979)  modification  are  both  rejected 
at  the  p<0.05  level  for  both  sets  of  data.  The  model  called  "ABCD"  is 
a  form  of  (2)  in  which  the  vector  5  is  estimated,  but  restricted  to  be 
equal  across  items:  this  represents  the  hypothesis  that  a  (constant) 
proportion  of  those  who  "don't  know"  select  alternative  A,  a  different 
proportion  B,  and  so  on;  but  the  distribution  over  A,  B,  C,  and  D 
(regardless  of  the  "correct"  response  alternative)  is  the  same  for  all 
items.  For  the  WK  data  the  model  is  a  great  improvement  over  Samejima's 
6=0  model  (G2=12.6  on  3  d.f.),  and  for  GS  it  is  not.  The  estimated  vec¬ 
tor  d  for  WK  for  [A,B,C,D]  is  [ . 1, .2, .4, .3] . 

In  the  last  four  WK  items,  D  is  correct  for  the  first  and  the  last 
and  C  is  correct  for  the  others;  so  to  test  the  hypothesis  that 
(somehow)  the  correct  alternative  "attracts"  guessing  we  tested 
"ABCD(C) ,ABGD(D)"  in  which  the  two  pairs  of  items  had  different  vectors 
6.  This  model  fits  the  WK  four-item  data,  and  the  improvement  over  ABCD 
is  significant  (G2=10.6,  d.f. =3,  p<0.02);  the  parameter  estimates  are 
given  in  table  3.  The  "saturated"  version  of  (2),  with  11  free 


Insert  Tables  3  and  4  About  Here 


The  last  four  GS  items  do  not  have  such  a  convenient  structure  of 
correct  alternatives;  but  for  these  items  "correctness"  seemed  more  im¬ 
portant  than  alternative  position,  so  in  the  model  "correct  vs.  incor¬ 
rect"  d^  is  constrained  to  be  one  value  (the  same  for  all  items)  for  the 
correct  response  and  another  for  all  the  incorrect  responses.  This 
model  (barely)  fits  the  GS  data.  The  parameter  estimates  are  given  in 
Table  4.  Forty  percent  of  the  DK  guess  the  correct  alternative;  there 
is  no  evidence  that  this  varies  across  items  (The  last  line  of  Table  2 
gives  the  G2  for  "correct  vs.  incorrect  each  item”). 

When  the  complete  WK  data  are  fitted,  the  likelihood  ratio  test 

against  a  "general  multinomial  alternative"  for  the  1178  observations 
35 

in  a  4  table  is,  of  course,  meaningless.  Nevertheless,  we  report 
likelihood  ratio  tests  between  variously  constrained  versions  of  the 
model.  The  Samejima  (1979)  model  is  the  most  constrained  form  that  we 
use  for  comparison.  A  model  of  the  "ABCD"  form  of  (2),  in  which  6  is 
estimated,  equal  across  items,  gives  a  reduction  G2=4l  on  3  d.f., 
p<0.001;  the  estimate  of  d  was  [ .2, .3, .3,  .2]  for  [A,B,C,D].  As  with  the 
four-item  WK,  a  model  in  which  6  is  constrained  to  be  equal  only  among 
items  with  the  same  (letter)  correct  response  fits  significantly  better: 
G2=72  on  9  d.f.,  p<0.01.  No  more  complex  models  were  considered,  as 
estimation  of  11  parameters  per  item  for  the  very  easy  items  (to  which 
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the  majority  of  examinees  responded  correctly)  would  be  estimating  many 
parameters  with  little  data.  The  estimated  parameters  for  the  last  four 
items  obtained  in  the  analysis  of  the  entire  WK  set  are  given  directly 
below  their  values  for  the  four-item  estimation  in  Table  3. 

Analysis  of  the  physical  science  portion  of  the  GS  section  gave 
similar  results:  (2)  in  its  ”ABCD”  form  fitted  significantly  better  than 
Samejima's  model,  G2=29  on  3  d.f.,  p<0.01.  The  estimate  of  d  was  again 
[ .2, .3, .3, .2] .  And  the  model  in  which  d  varied  according  to  the  letter- 
value  of  the  correct  alternative  was  better:  G2  =  26  on  9  d.f.,  p<0.01. 
Two  of  the  items  in  the  ’’physical  science”  set  are  among  the  last  four 
GS  items,  and  their  parameter  estimates  are  given  in  Table  4. 

Discussion 

In  cases  in  which  the  fit  can  be  tested  —  with  a  thousand  ex¬ 
aminees  and  four-alternative  items,  four  items  —  some  forms  of  the 
model  given  by  (2)  and  (5)  fit  item  response  data  satisfactorily.  This 
represents  a  major  Step  forward  in  item  analysis,  because  it  is  no 
longer  necessary  to  look  at  deviations  from  the  fit  to  examine  items; 
we  may  examine  the  fit  itself. 


Insert  Figure  1  About  Here 


The  solid  curves  in  Figure  1  illustrate  the  trace  lines  for  the 
"physical  science"  fit  to  GS  item  23,  which  is  item  2  in  Table  1, 
discussed  above.  From  Table  1  we  inferred  that  those  who  '■ho  '1  A  or  3 
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might  be  more  able  than  those  who  chose  D.  Indeed,  the  trace  line  for 
D  restricts  that  response  to  thos"'  with  values  of  6<0  for  all  practical 
purposes,  while  B-responses  come  mostly  from  those  with  9>G,  and  A  is 
spread  all  over.  The  DK  trace  line  indicates  that  most  of  those  below 
9=-. 5  have  no  idea;  and  more  of  them  guess  correctly  than  any  other  way. 

The  dashed  curves  in  Figure  1  illustrate  the  trace  lines  for  the 
"four-item"  fit  to  item  23.  Note  in  Table  4  that  the  parameter 
estimates  differ  a  great  deal,  but  the  solid  and  dashed  trace  lines  in 
Figure  1  differ  much  less  (except  for  DK)  in  the  middle .  DK  is  a  com¬ 
pletely  inferred  latent  response  and  is  not  really  very  well-defined  by 
four  items.  The  other  curves  are  more  similar  in  the  middle,  where  the 

data  are;  95%  of  the  population  distribution  lies  between  0s  of  -2  and 

* 

+2.  The  different  parameter  estimates  seem  to  affect  the  curves 
primarily  at  the  extremes. 

Unlike  algorthms  which  make  use  of  point-estimates  of  9  in  the 
estimation  of  item  parameters,  and  require  both  large  numbers  of  ex¬ 
aminees  and  large  numbers  of  items,  the  MMX  estimation  procedure  used 
here  should  be  consistent  considering  the  number  of  examinees  alone  if 
the  model  is  correct.  That  is,  as  the  number  of  examinees  becomes  large 
the  estimates  for  four  items  or  12  or  35  should  all  converge  to  the  true 
values.  So  the  dashed  and  solid  lines  in  Figure  1  should  be  the  same. 
They  are  very  similar,  and  there  are  three  possible  reasons  for  the 
small  differences  observed  between  them.  First,  one  thousand  examinees 
may  easily  not  be  ''asymptotic"  for  a  table  with  256  cells.  Second,  one 
or  the  other  solution  may  not  be  completely  converged;  EM-algorithms  can 
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be  slow  near  the  maximum.  Third,  and  most  likely,  the  model  may  be  in¬ 
correct  in  that  the  latent  dimension  S  may  be  defined  slightly  dif¬ 
ferently  by  the  most  difficult  four  items  than  by  the  entire  test. 


Insert  Figure  2  About  Here 


Figure  2  similarly  illustrates  the  four-item  and  complete  test 
estimates  of  the  trace  lines  for  WK  32,  which  demonstrates  this  effect 
more  graphically.  With  estimates  from  the  entire  test,  DK  goes  to  unity 
for  low  9.  But  the  four-item  (dashed)  curve  for  DK  rises  for  moderately 
low  9  and  then  goes  back  down,  and  A  goes  to  unity  for  low  9.  That 
can't  be  right;  but  it  is  all  happening  in  the  region  below  S=-2,  where 
there  are  essentially  no  data  and  the  model  is  extrapolating.  Ex¬ 
trapolation  is  bad:  the  model  is  too  flexible  at  the  extremes.  Flex¬ 
ibility  was  one  of  the  goals  of  the  model,  but  it  seems  we  may  have 
overdone  it.  This  is  in  marked  contrast  to  traditional  IRT  models  which 
go  to  unity,  zero,  or  some  asymptote  at  the  extremes,  and  never  mis¬ 
behave  there.  Note  that  even  with  all  of  this  strangeness,  the  other 
trace  lines  in  Figure  2  are  essentially  identical  between  9s  of  -2  and 
+2. 


Insert  Figure  3  About  Here 


Figure  3  shows  the  trace  lines  for  the  correct  responses  only  for 
WK  items  32-35  from  the  four  item  set  and  the  whole  test,  as  well  as 
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traditional  3-PL  curves  estimated  for  the  entire  test  (dotted  lines)  for 
comparison.  If  the  "tail-wagging"  to  the  left  of  0=-2  is  ignored,  the 
pairs  of  curves  for  the  whole  test  are  nearly  identical  and  the  four- 
item  curves  are  somewhat  deviant.  That  may  be  capitalization  on  chance 
in  four  items  —  or  it  may  be  that  0  differs  mildly  in  the  hardest  items 
of  the  test  from  its  definition  in  the  entire  test.  None  of  the  curves 
are  extremely  different. 


Insert  Figure  4  About  Here 


For  WK  as  a  whole,  the  usual  result  (of  Bock,  1972;  Thissen,  1976) 
is  obtained  with  respect  to  test  information:  Figure  4  shows  the  infor¬ 
mation  curves  for  the  multiple  category  scoring  and  the  binary  scoring 
(3-PL).  For  0<O,  information  from  incorrect  responses  increases  total 
information  by  about  50%,  equivalent  to  extending  the  test  (for  half  the 
examinees)  from  35  to  50  items. 


Insert  Figure  5  About  Here 


Figure  5  illustrates  the  process.  The  solid  curves  show  the  trace 
lines  associated  with  a  particular  set  of  responses  [B,B,B,B],  all  in¬ 
correct,  for  the  last  four  WK  items.  Then  the  N(0,1)  population 
distribution  is  plotted  with  the  the  product  of  all  five  curves,  or 
posterior  density,  labelled  "Total."  The  mode  of  that  density  is 
traditionally  0;  it  is  about  -0.7  (s.e.=0.6)  in  this  case.  The  dashed 
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curves  show  the  3-PL  trace  lines,  all  incorrect,  so  they  are  all 
monotonic  decreasing.  That  is  not  very  informative;  the  (dashed) 
posterior  is  broader  (s.e.=0.7)  and  closer  to  zero  (6=-0.5)  because 
there  is  less  information  so  the  estimator  is  "shrunk”  more  toward  the 
mean  of  the  population  distribution.  If  the  model  approximates  the 
world  reasonably  well,  the  multiple  category  scoring  gives  more  informa¬ 
tion  about  people  responding  [BBBB]  than  can  3-PI. 

General  Discussion 

The  model  proposed  here  provides  the  first  practical,  complete  IRT 
item  analysis  for  multiple  choice  tests,  describing  the  performance  of 
all  of  the  response  alternatives  as  functions  of  the  trait  being 

* 

measured.  The  parameters  of  the  model  are  not  readily  interpretable  for 
the  most  part,  but  graphical  presentation  of  the  trace  lines  and/or 
their  products  with  the  population  distribution  gives  thorough  item 
analysis  which  has  much  to  recommend  it. 

This  sort  of  item  analysis  is  sufficiently  flexible  that  "bad" 
trace  lines  are  fitted.  These  can  then  be  observed  and  such  items 
modified  or  eliminated  in  the  course  of  empirical  test  development. 

That  is  not  possible,  in  general,  with  simpler  models  which  require  ex¬ 
amination  of  the  residuals  to  find  bad  items.  For  item  analysis,  the 
•model  of  (2)  and  (5)  is  clearly  an  excellent  choice;  for  scoring  tests 

a 

(aka  "estimating  9"),  matters  are  more  complicated. 

Non-monotonic  trace  lines  for  the  correct  response  have  become  a 
popular  feature  of  recent  IRT  developments:  Lord  (1982)  and  Choppin 
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(1983)  have  proposed  item  response  models  which  permit  "low-ability" 
non-monotonicity,  in  work  on  binary  models  independent  from  Samejima's 
(1979)  multiple  category  proposal.  There  are  a  number  of  possible  ex¬ 
planations  for  non-monotonicity  on  the  left,  as  for  the  correct  response 
in  Figure  1.  The  only  requirement  for  candidacy  as  an  explanation  for 
the  effect  is  that  it  must  account  for  "getting  the  right  answer  for  the 
wrong  reason."  Bock  (personal  communication,  1983)  suggests  a  name  for 
the  phenomenon:  "positive  misinformation." 

Two  sources  of  positive  misinformation  come  to  mind.  The  first 
is  that  the  correct  response  for  a  particular  item  differs  from  the 
distractors  on  dimensions  other  than  that  which  is  intended  and  ob¬ 
servable,  given  sufficient  ability.  Examinees  of  medium  and  high 
ability  perceive  the  features  and  attempt  the  processing  intended  by  the 
item-writer,  while  examinees  of  low  ability  see  other  features  of  the 
alternatives  which  cause  them  to  select  the  correct  one.  These  "other 
features"  may  be  effectively  "invisible"  to  individuals  of  higher 
ability,  and  therefore  to  the  item-writers  as  well. 

A  second  possible  cause  of  non-monotonicity  in  the  correct  trace 
line  is  cheating.  If  low-ability  individuals  cheat  (e.g.,  by  copying 
a  neighbor's  answer,  which  is  more  likely  to  be  correct  than  not  for 
most  items),  then  the  resulting  correct  alternative  trace  lines  will 
rise  for  those  of  very  low  ability. 

It  may  also  be  possible  that  non-monotonic  trace  lines  are  estima¬ 
tion  artifacts  in  small  sets  of  items  (like  four).  It  may  be  that,  with 
few  monotonic  correct-alternative  trace  lines  to  "orient"  9,  the  estima- 
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high  ability  individuals  on  the  left  end  of  the  9-continuum.  But  this 
explanation  can  be  discounted  with  longer  tests,  as  the  non-monotonicity 
there  comes  from  correct  response  to  an  item  being  more  likely  from  ex¬ 
aminees  who  get  most  of  the  other  items  wrong  than  from  those  who  get 
fewer  of  the  other  items  wrong. 

Distinguishing  among  these  possibilities  would  require  experimen¬ 
tation  with  the  items  and  the  testing  situation.  The  model  does, 
however,  offer  an  item  analysis  which  permits  such  phenomena  to  exhibit 
themselves  in  the  fit,  when  they  are  present. 

So  the  use  of  non-monotonic  curves  in  item  analysis  may  be  re¬ 
quired  to  fit  the  observed  data.  However,  the  use  of  noc-monotonic 
trace  lines  in  constructing  the  posterior  density,  a  measure  of  the  cen- 

a 

tral  tendency  of  which  is  to  be  called  0  and  used  to  "score  the  test", 
gives  rise  to  many  potential  problems.  If  the  correct  response  trace 
line  for  item  j  is  non-monotonic  on  the  right  (e.g.,  it  turns  down), 
then  examinees  with  some  response  patterns  on  the  other  items  will  be 
"penalized"  by  responding  correctly  to  item  j;  they  would  have  been  as- 

A 

signed  a  higher  0  if  they  had  selected  certain  of  the  incorrect  alterna¬ 
tives.  Correct  response  trace  lines  which  are  non-mono tonic  on  the  left 
similarly  "penalize"  examinees  of  low  ability  who  respond  correctly.  As 
measurement,  this  is  all  probably  satisfactory:  conditional  on  the 
other  item  responses ,  a  correct  response  to  item  j  may  not  imply  higher 
ability,  it  may  be  more  likely  to  be  guessing  or  cheating.  But  this  may 
be  a  problem  for  the  test  "as  contest."  Further,  it  might  be  difficult 
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to  defend  such  a  method  of  test  scoring  against  the  onslaught  of  members 
of  the  bar  before  a  jury.  A  case  can,  therefore,  be  made  for  either  a) 
rejecting  the  model  or  b)  rejecting  the  use  of  items  with  non-monotonic 
trace  lines  for  the  correct  response. 

Multiple  category  scoring  of  this  nonlinear  sort  has  certain 
bizarre  properties  even  if  non-monotonic  trace  lines  are  eliminated. 

For  certain  regions  on  the  6-continuum,  selecting  a  particular  incorrect 
response  will  increase  8  more  than  would  selecting  the  actual  correct 
response.  For  instance,  in  the  Word  Knowledge  item  in  Figure  2,  for  0 

a 

just  below  -1,  selecting  response  B  will  increase  6  more  than  would 
selecting  the  correct  response.  But  for  9  around  1,  selecting  3  will 
"penalize"  the  respondent!  The  contingencies  are  sufficiently  complex 
that  it  is  unlikely  that  the  examinees  could  find  a  strategy  to  take  ad¬ 
vantage  of  the  system.  But  the  possibilities  for  legal  difficulties  are 
considerable  under  circumstances  in  which  the  test  and  scoring  system 
must  be  disclosed.  On  the  other  hand,  the  quality  of  measurement  for 
research  purposes  and  in  non-disclosed  tests  should  be  improved  by  these 
"bizarre"  features;  that  is,  after  all,  where  the  multiple  category 
model  obtains  its  additional  information. 

Conclusion 

Item  response  models  for  multiple  choice  items  have  come  of  age: 
they  fit  the  data.  Questions  remain  about  uses  for  these  models.  They 
produce  excellent  item  analysis,  but  it  is  complex  and  best-represented 
graphically  —  that  breaks  with  the  tradition  of  item  analysis  with  a 
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few  numerical  summaries.  Multiple  category  scoring  clearly  increases 
the  information  obtained  in  scoring  a  test  —  but  again  at  the  cost  of 
complexity,  and,  potentially,  controversy.  The  validity  of  test  scored 
with  such  methods  has  not  been  examined  here,  nor  can  it  be  with  inter¬ 
nal  consistency  data  in  any  event.  Consideration  of  all  of  these 
matters  is  deferred  to  future  work. 
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Table  1. 

Cross  classification  of  responses  to  two  items  (22  and  23) 
of  the  ASVAB  Form  8A  General  Science  subtest  for  1043  examinees 
of  the  NORC  national  probability  sample  (Bock  and  Mislevy,  1981). 

Item  2 


A  B  C*  D 


A 

20(22) 

107(23) 

79(22) 

44(31) 

250 

Item 

B* 

32(36) 

157(34) 

143(40) 

35(25) 

367 

1 

C 

7(  8) 

36 (  8) 

24(  7) 

19(14) 

86 

D 

30(33) 

161(35) 

112(31) 

42(30) 

345 

89 

461 

358 

140 

1048 

*=Correct; 

Column  percentages  in  parentheses. 
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Table  2. 

G2  values  for  selected  models  for  the 
four-item  examples. 


Word 

General 

Knowledge 

Science 

Model 

d.f. 

G2 

P 

G2 

P 

Bock  (1972) 

231 

307 

<.01 

277 

<.02 

Samejima  (1979) 

223 

271 

<.02 

264 

<.05 

"ABCD" 

220 

258 

<.04 

262 

<.02 

”ABCD(C),ABCD(D)" 

217 

248 

>.05 

— 

— 

"11  per  item" 

211 

245 

<.05 

— 

-- 

"Correct  vs.  incorrect" 

222 

— 

— 

253 

II 

o 

cn 

"Correct  vs.,  each  item" 

3 

— 

-- 

253 

=  .05 
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Table  3. 

Estimated  parameters  for  the  last  four  Word  Knowledge  items. 
Estimates  for  four  items  above,  whole  test  below. 
Parameters  for  correct  response  underscored. 


Response 


DK 

A 

B 

C 

D 

-1.1 

-2.9 

1.5 

0.2 

2.2 

-1.7 

-1.0 

1.1 

0.3 

1.9 

0.1 

-3.8 

2.9 

-1.9 

2.7 

0.3 

-2.3 

2.4 

-2.5 

2.1 

0.1 

0.2 

0.3 

0.4 

0.25 

0.25 

0.25 

0.25 

-3.0 

-0.3 

1.2 

2.8 

-0.7 

-2.1 

-0.6 

1.2 

2.3 

-0.8 

0.1 

Q  .6 

-1.6 

0.1 

0.8 

2.1 

0.5 

-3.0 

-O 

1.0 

0.2 

0.2 

0.5 

0.1 

0.2 

0.2 

0.4 

0 . 2 

-0.8 

-1.8 

-0.1 

1.9 

0.7 

-1.3 

-0.9 

-0.2 

1.9 

0.5 

-1.6 

-3.2 

0.2 

2.6 

2.0 

-0.9 

-2.5 

-0.1 

1.8 

1.6 

0.2 

0.2 

0.5 

0.1 

0.2 

0.2 

0.4 

0.2 

-1.6 

-0.4 

-0.2 

-0.8 

3.0 

-1.9 

0.5 

0.0 

-0.6 

1.9 

0.0 

-1.3 

0.4 

0.4 

0.5 

-0.1 

-2.0 

0.5 

0.8 

0.8 

0.1 

0.2 

0.3 

0.4 

0.25 

0.25 

0.25 

0.25 

.  *  •  .  v^  ,*•  .  '  . 
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Table  4. 

Estimated  parameters  for  the  last  four  General  Science  items. 
Estimates  for  four  items  above,  "physical  science"  subset  below. 


Parameters  for  correct  response  underscored. 
Response 
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Figure  Captions 

Figure  1.  Five  panels  showing  P(k),  k=0,l,...,m.  as  a  function  of 
Q;  the  solid  lines  correspond  to  the  probabilities  for  DK,  and  responses 
A,B,C,  and  D  for  GS  item  23,  estimated  as  part  of  "physica1  science." 

The  dashed  curves  use  the  estimates  obtained  with  the  last  four  items. 

Figure  2.  Five  panels  showing  P(k),  k=0 ,  1 , . . .  ,ak  as  a  function  of 
9;  the  solid  lines  correspond  to  the  probabilities  for  DK,  and  responses 
A,B,C,  and  D  for  WK  item  32  estimated  in  the  entire  test.  The  dashed 
curves  use  the  estimates  obtained  with  the  last  four  items. 

Figure  3.  Four  panels  show  three  alternative  fits  of  the  trace 
line  for  the  correct  responses  for  the  last  four  WK  items:  dots  are  on 
the  standard  3-PL  curve,  the  solid  line  is  the  model  of  the  present 
paper  fitted  with  the  entire  test,  and  the  dashed  line  is  the  present 
model  fitted  with  only  those  items. 

Figure  4.  Test  information  curves  for  binary  (dotted  lin-*)  and 
multiple  category  (solid  line)  scoring  for  WK,  as  functions  of  9. 

Figure  5.  The  solid  lines  give  the  trace  lines  from  the  present 
model  and  posterior  density  (labelled  "Total")  for  response  pattern 
[BBBB]  (all  incorrect)  for  the  last  four  items  of  WK.-  Dashed  lines  give 
the  corresponding  curves  for  the  same  response  pattern  using  3-PL 
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Dr.  Scott  Maxwell 
Department  of  Psychology 
University  of  Notre  Dame 
Notre  Oame,  IN  46556 


Or.  Samuel  T.  Mayo 
Loyola  University  of  Chicago 
820  North  Michigan  Avenue 
Chicago,  IL  60611 


Mr.  Robert  McKinley 
American  College  Testing  Pro. 
P.0.  Box  168 
Iowa  City,  IA  52243 


Or.  Barbara  Means 
Human  Resources  Research  Organ. 
300  North  Washington 
Alexandria,  VA  22314 

Or.  Allen  Munro 
Behavioral  Tech.  Laboratories 
1845  Elena  Ave. ,  Fourth  Floor 
Redondo  Beach,  CA  90277 


Or.  W.  Alan  Nicewander 
University  of  Oklahoma 
Department  of  Psychology 
Oklahoma  City,  OK  73069 


Or.  Oonald  A.  Norman 
Cognitive  Science,  C-015 
Unt'v.  of  California,  San  Diego 
La  Jolla,  CA  92093 


Or.  Melvin  R .  Nov i ck 
356  Lindquist  Cnt.  for  Measure. 
Un i vers i ty  of  I owa 
Iowa  City,  I A  52242 


Dr.  James  Olson 
WICAT,  Inc. 

1875  Wouth  State  Street 
Orem,  UT  84057 

Wayne  M.  Patience 
American  Council  on  Education 
GED  Testing  Service,  Suite  20 
One  Oupon  Cirl'fe,  NW 
Washington,  DC  20036 


Dr.  James  A.  Paulson 
Portland  State  University 
P.0.  Box  751 
Portland,  OR  97207 


Dr.  James  W.  Pellegrino 
University  of  California, 
Santa  Barbara 
Oeoartment  of  Psychology 
Santa  Barabara,  CA  93106 

Or.  Mark  D.  Reckase 
ACT 

P.0.  Box  168 
Iowa  City,  I A  52243 


Dr.  Lauren  Resnick 
LRDC 

University  of  Pittsburgh 
3939  O'Hara  Street 
Pi ttsburg ,  PA  1 5260 

Dr.  Thomas  Reynolds 
University  of  Texas-Dallas 
Marketing  Department 
P.0.  Box  688 
Richardson,  TX  75080 

Dr.  Andrew  M.  Rose 
A mer.  Insti.  for  Research 
1055  Thomas  Jefferson  St.  NW 
Washington,  DC  20007 


Dr.  Lawrence  Rudner 
403  Elm  Avenue 
Takoma  Park,  M0  20012 


Dr.  J.  Ryan 

Oeoartment  of  Education 
Univeristy  of  South  Carolina 
Columbia,  SC  29208 


Prof.  Fumiko  Samejima 
Department  of  Psychology 
Univeristy  of  Tennessee 
Knoxvi lie,  TN  37916 


Dr.  Walter  Schneider 
Psychology  Department 
603  E.  Daniel 
Champaign,  IL  6 1 320 

Dr.  Kazuo  Shigemasu 
7_9“24  Kugenuma-Ka i gan 
Fujusawa  251 
JAPAN 


Or..  Erling  8.  Andersen 
Department  of  Statistics 
Studiestraede  6 
1-455  Copenhagen 
DENMARK 

Dr.  Isaac  8ejar 
Educational  Testing  Service 
Princeton,  NJ  08450 


Dr..  Menucha  Sirenbaum 
School  of  Education 
Te.il  Aviv  University 
Te4  Aviv,  Ramt  Aviv  60078 
ISRAEL 

Dr,  Werner  8  i  rke 
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Or,  Robert  Brennan 
American  Col iege-Testing  Pro¬ 
grams 

P.0.  Box  168 
Iowa  City,  I A  52243 

Or,  Glenn  Bryan 
6208  Poe  Road 
£e£hes da ,  M0  20817 


Or.  Ernest  R.  Cadotte 
307  Stoke ly 

University  of  Tennnessee 
Knoxville,  TN  37916 


[;•  Dr,  John  B.  Carroll 
f-'  409  Elliott  Road 

u  Chapel  Hill  ,  NC  27514 


Dr,  Norman  Cliff 
Department  of  Psychology 
Univ.  of  So.  California 
Univesity  Park 
Las  Angeles,  CA  90007 

Dr.  Lloyd  Humphreys 
Department  of  Psychology 
Un.i  vers  i  ty  of  Illinois 
603  East  Oaniel  Street 
Champaign,  II  6 1820 

Dr,  William  Koch 
University  of  Texas -Austin 
Measurement  and  Evaluation 

Center 

Austin,  TX  78703 


Or.  Allan  M.  Collins 
Bolt  Beranek  &  Newman,  Inc. 
50  Moulton  Street. 
Cambridge,  MA  02138 


Or.  Hans  Crombag 
Education  Research  Center 
University  of  Leyden 
Boerhaavelaan  2 
2334  EN  Leyden,  NETHERLANDS 

CTB/McGraw-Hi 1 1  Library 
2500  Garden  Road 
Monterey,  CA  93940 


Dr.  Oattpradad  Divgi 
Syracuse  University 
Department  of  Psychology 
Syracuse,  NE  33210 


Or.  Fritz  Orasgow 
Department  of  Psychology 
Univeristy  of  Illinois 
603  E.  Oaniel  St. 
Champaign,  IL  6 1 820 

Or.  Susan  Embertson 
Psychology  Department 
University  of  Kansas 
Lawrence r  KS  6604$ 


ERIC  Facility-Acquisitions 
4833  Rugby  Avenue 
Bethesda,  M0  20014 


Or.  Benjamin  A.  Fairbank,  Jr. 
McFann-Gray  5  Associates,  Inc. 
5825  Callaghan 
Suite  225 

San  Antonio,  TX  78228 

Dr.  Leonard  Feldt 
Lindquist  Center  for  Measure¬ 
ment 

Univesity  of  Iowa 
Iowa  City,  I A  52242 

Or.  Earl  Hunt 
Department  of  Psychology 
University  of  Washington 
Seattle,  WA  98105 


Or.  Jack  Hunter 
2122  Cool i dge  Street 
Lansing,  Ml  48906 


Dr.  Richard  L.  Ferguson 
The  American  College  Testing 
Program 

P.0.  Box  168 
Iowa  Ci ty  ,  I A  52240 

Univ.  Prof.  Dr.  Gerhard 
Fi scher 

Liebiggasse  5/3 
A  1010  Vienna 
AUSTRIA 

Professor  Oonald  Fitzgerald 
University  of  New  England 
Arm i dale.  New  South  Wales 
2351 

AUSTRALIA 

Dr.  Dexter  Fletcher 
University  of  Oregon 
Department  of  Computer  Science 
Eugene,  OR  97403 


Or.  John  R.  Rfederiksen 
Bolt  8eranek  &  Newman 
50  Moulton  Street 
Cambridge,  MA  02138 


Dr.  Janice  Gi fford 
University  of  Massachusetts 
School  of  Education 
Amherst,  MA  07002 


Dr.  Robert  Glaser 
Learning  Research  &  Dev.  Cente 
University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh,  PA  15260 

Dr.  Bert  Green 
Johns  Hook  ins  University 
Department  of  Psychology 
Charles  £  34th  Street 
Baltimore,  M0  21218 

Or.  Ron  Hambleton 
School  of  Education 
University  of  Massachusetts 
Amherst,  MA  01002 


Or.  Paul  Horst 
677  G  Street,  ?184 
Chula  Vista,  CA  90010 


Dr.  Huynh  Huynh 
College  of  Education 
University  of  South  Carolina 
Columbia,  SC  29208 
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AIR  FORCE  HUMAN  RESOURCES  LA8 
AFHRL/MPO 

BROOKS  AFB ,  TX  78235 


AIR  UNIVERSITY  LIBRARY 

AUL/LSE  76/443 
MAXWELL  AFB,  AL  36112 


UR.  ALFRED  R.  FREGLY 
ATOSR/NL 

BOLLING  AFB,  DC  20332 


MR.  RANDOLPH  PARK 
AFHRL/MOAN 

BROOKS  AFB,  TX  78235 


3700  TCHTW/TTGHR 
2LT  TALLARIGO 
SHEPPARD  AFB,  TX  76311 


OR.  JOSEPH  YASATUKE 
>  AFHRL/LRT 
UDWRY  AFB,  CO  80230 


OR.  WILLIAM  GRAHAM 
TESTING  DIRECTORATE 
MEPCOM/MEPCT-P 
FT.  SHERIDAN,  IL  60037 


OR.  WAYNE  SELLMAN 
OFF.  OF  THE  ASS.  SEC.  OF 
DEFENSE  (MRA  &  L) 

2B269  the  Pentagon 
WASHINGTON,  DC  20301 

OR.  ARTHUR  MELMEO 
724  BROWN 

US  DEPARTMENT  OF  ED. 
WASHINGTON,  OC  20208 


MR.  THOMAS  A.  WARM 
US  COAST  GUARD  INST. 
P.O.  SUBSTATION  18 
OKLAHOMA  CITY,  OK  73169 


Technical  Documents  Center 
AF  Human  Resources  Lab 
WPAFB ,  OH  45433 


OR.  EARL  A.  ALLUISI 
HQ,  AFHRL  (AFSC) 
BROOKS  AFB,  TX  78235 


OR.  GENEVIEVE  HADDAD 
PROGRAM  MANAGER 
LIFE  SCIENCES  DIRECTORATE 
AFOSR 

BOLLING  AFB,  DC  20332 

OR.  ROGER  PENNELL 
AF  HUMAN  RESOURCES  LAB 
LOWRY  AF8 ,  CO  80230 


LT.  COL  JAMES  £.  WATSON 
HQ  USAF/MPXOA 
THE  PENTAGON 
WASHINGTON,  OC  20330 


OEFENSE  TECHNICAL  INFO  CEN. 
CAMERON  STATION,  BLOG  5 
ALEXANDRIA,  VA  22314 
ATTN :  TC 


Jerry  Lehnus 
HQ  MEPCOM 
Attn:  MEPCT-P 
Fort  Sheridan,  II  60037 


MAJOR  JACK  THORPE 
OARPA 

1400  WILSON  BLVO. 
ARLINGTON,  VA  22209 


OR.  ANOREW  R.  MOLNAR 
OFF  OF  SCI  6  ENG  PER  £  ED 
NAT'L  SCIENCE  FOUNDATION 
WASHINGTON,  DC  2041 5 


OR.  JOSEPH  L.  YOUNG,  DIR. 
MEMORY  5  COGNITIVE  PROCESSES 
NAT'L  SCIENCE  FOUNDATION 
WASHINGTON,  DC  20550 


US  AF  OFFICE  OF  SCI  .  RES.  -A 

LIFE  SCI .  DIRECTORATE,  NL 
BOLLING  AIR  FORCE  BASE 
WASHINGTON,  OC  20332 


MR.  RAYMOND  E.  CHRISTAL 
AFHRL/ MOE 

BROOKS  AFB,  TX  ?823$ 


OR.  T.  M.  LONGRIDGE 

AFHRL/ OTE  TiB 

WILLIAMS  AFB,  AZ  85224 


DR.  MALCOM  REE 
AFHRL/ MP 

BROOKS  AFB,  TX  78235 


MAJOR  JOHN  WELSH 
AFHRL/MOAN 
BROOKS  AFB,  TX 


DR.  CRAIG  I .  FIELDS 
ADVANCED  RES.  PROJECTS  AGENCY 
J 400  WILSON  BLVD. 

ARLINGTON,  VA  22209 


.  *  .  « 


M I  L I  TARY  ASS  .  FOR  TRA I N  I NG  £  /-'?■ 
PERSONNEL  TECH. 

OFF.  OF  THE  UNOER  SEC.  FOR  RE^ 
ROOM  30129,  the  PENTAGON  “ 
WASHINGTON,  OC  20301 


DR.  SUSAN  CHIPMAN 
LEARNING  £  DEVELOPMENT 
NAT'L  INST.  OF  EDUCATION 
1200  19th  STREET  NW 
WASHINGTON,  DC  20208 


DR.  VERN  W.  URRY 
PERSONNEL  R£0  CENTER 
OFF  OF  PERSONNEL  MANAGEMENT 
1900  E  STREET  NW 
WASHINGTON,  DC  20415 


DR.  HAMES  ALGINA 
UNIVERSITY  OF  FLORIDA 
GAINESVILLE,  FI  326 


MR.  BRAD  SYMPSOM 

NAVY  PERSONNEL  R£D  CENTER 

SAN  01  EGO ,  CA  92152 


OR.  MARTIN  A.  TOLCOTT 
ASS.  OIR.  FOR  LIFE  SCIENCE 
800  NORTH  QUINCY  STREET 
ARLINGTON,  VA  22217 


OR.  JAMES  TWEEOOALE 
TECHNICAL  01  RECTOR 
NAVY  PERSONNEL  R£0  CENTER 
SAN  DIEGO,  CA  92152 


PERSONNEL  £  TRAINING 
RESEARCH  GROUP 
COOE  UA2PT 

OFFICE  OF  NAVAL  RESEARCH 
ARLINGTON,  VA  22217 

IT  FRANK  C.  PETHO,  MSC, 

USN  (PH. 0) 

CNET  (N-432) 

NAS 

PENSACOLA,  FL  32508 

OR.  CARL  ROSS 
CNET-POCD 
.BUILDING  90 

GREAT  LAKES  NTC ,  IL  60088 

OR-  R.  SNOW,  LIAISON  SCI . 
OFFICE  OF  NAVAL  RESEARCH 
BRANCH  OFFICE,  LONDON 
BOX  39 

FPO  NEW  YORK,  NY  09510 

OR.  FRANK  VI  Cl  NO 

NAVY  PERSONNEL  RfiO  CENTER 

SAN  01  EGO ,  CA  92152 


SPECIAL  ASST.  FOR  EDUCATION 
ANO  TRAINING  (OP-OIE) 

RM.  2705  ARLINGTON  ANNEX 
WASHINGTON,  OC  20370 

OFFICE  OF  THE  CHIEF  OF  NAVAL 
OPERATIONS 

RES.  OEV .  £  STUDIES  8 RANCH 
OP  115 

WASHINGTON,  OC  20350 

OR.  ROBERT  G.  SMITH 

OFFICE  OF  CHIEF  OF  NAVAL  OP. 

0P-987H 

WASHINGTON,  OC  20350 


OR.  RICHARO  SORENSEN 
NAVY  PERSONNEL  R£0  CENTER 
SAN  01  EGO.  CA  92152 


OR.  EOUARO  WEGMAN 
OFFICE  OF  NAVAL  RESEARCH 
800  NORTH  QUINCY  STREET 
ARLINGTON,  VA  22217 
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OR.  BERNARO  RIMLANO  (01 C) 
NAVY  PERSONNEL  R&D  CENTER 
SAN  DIEGO,  CA  95152 


OR.  ALFRED  F.  SMODE,  DIR. 
TRAINING  ANALYSIS  £ 
EVALUATION  GROUP 
OEPT.  OF  THE  NAVY 
ORLANDO,  FL  32813 

OR.  FREDERICK  ST-E I NHE I SER 
CNO  -  OP  115 
NAVY  ANNEX 

ARLINGTON,  VA  20370 


DR.  RONALD  WEITZMAN 
NAVAL  POSTGRADUATE  SCHOOL 
OEPT.  OF  ADM.  SCIENCES 
MONTEREY ,  CA  939^0 


OR.  DOUGLAS  WETZEL 
COOE  12 

NAVY  PERSONNEL  R£0  CENTER 
BAN  DIEGO,  CA  92152 


OR.  MARTIN  F.  WISKOFF 
NAVY  PERSONNEL  RSO  CENTER 
SAN  01  EGO,  CA  92152 


MR.  JOHN  H.  WOLFE 

NAVY  PERSONNEL  R£D  CENTER 

SAN  DIEGO,  CA  92152 


DR.  WALLACE  WULFECK,  I  I  I 
NAVY  PERSONNEL  R£0  CENTER 
SAN  01  EGO,  CA  92152 


OR.  ROBERT  CARROLL 
NAVOP  115 

WASHINGTON,  OC  20370 


OR.  STANLEY  COLLYER 
OFFICE  OF  NAVAL  TECHNOLOGY 
300  N.  QUINCY  STREET 
ARLINGTON,  VA  22217 


OR.  RICHARO  SLSTER 
OEPT.  OF  AOM.  SCIENCES 
NAVAL  POSTGRADUATE  SCHOOL 
MONTEREY,  CA  939^0 


OR.  NICK  BONO 
OFFICE  OF  NAVAL  RESEARCH 
LIAISON  OFFICE,  far  EAST 
APO  SAN  FRANCISCO,  CA  9^503 


OR.  ROBERT  3REAUX 
NAVTRAEQU1PCEN 

COOE  N-Q95R 
QRLANOO,  FL  32813 


CDR  MIKE  CURRAN 
OFFICE  OF  NAVAL  RESEARCH 
800  N.  QUINCY  ST. 

COOE  270 

ARLINGTON,  VA  222 '7 

DR.  PAT  FEDERICO 
COOE  PI  3 
NPROC 

SAN  01  EGO,  CA  92152 


LT.  ALEXANOER  BORY 
applied  psychology 
measurement  01  vis  I  on 

NAMRL 

NAS  PENSACOLA,  FL  32508 

Chief  of  Naval  Ed.  £  Train  Inc 
Liason  Office 

Ai r  Force  Human  Resource  Lao 
Operations  Training  Division 
WILLIAMS  AF3,  AZ  85224 

MIKE  ourmeyer,  INSTRUCTIONAL 
PROGRAM  DEVELOPMENT 
BUILDING  90 
NET-POCD 

GREAT  LAKES  NTC,  IL  60088 

DR.  CATHY  FERNANDES 
NAVY  PERSONNEL  R£0  CENTER 
SAN  DIEGO,  CA  92152 
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MR.  RAUL  FOLEY 

MAW  PERSONNEL  RSO  CENTER 

5 AN  DISCO.  CA  92152 


OR.  £0  HUT :n INS 

NAVY  PERSONNEL  RSO  CENTER 

SAM  niEGQ,  CA  9213?. 


OR.  VILLI  AH  L.  MALOY  (20) 
CHIEF  OF  NAVAL  ED.  S 
TRAINING 

NAVAL  AIR  STATION 
PENSACOLA,  FL  32508 

DR.  WILLIAM  E.  NOROBROCK 
"MC-AOCO  BOX  25 
APO.  NY  09/10 


WILL JAM  GREENUP 
EDUCATION  ADVISOR  (E031) 
EDUCATION  CENTER,  MCDEC 
QUANT  I  CO,  VA  22134 


S^EC.  ASS.  FOR  MARINE  MATTERS 
CODE  100M 

OFFICE  OF  NAVAL  RESEARCH 
8D0  N.  QUINCY  STREET 
ARLINGTON,  VA  22217 

TECHNICAL  01  RECTO? 

U.~S.  ARMY  RESEARCH  INST. 
BEHAVOP.IAL  S  SOCIAL  SCIENCES 
5001  EISENHOWER  AVENUE 
ALEXANDRIA,  VA  22333 


OR.  BEATRICE  J.  FARR 
a.  3.  ARMY  RES  E/ARCH  INST. 
5001  EISENHOL'SR  AVENUE 
ALEXANOR I A ,  VA  22333 


TP.  HAROLD  F.  O'NEIL,  JR. 
DIRECTOR.  TRAINING  RES.  LA8 
ARMY  RESEARCH  INSTITUTE 
5001  EISENHOWER  AVENUE 
ALEXANDRIA,  VA  22333 

MR-  ROB E FT  ROSS 

US  ARMY  RESEARCH  INST.  FOR  THE 
SOCIAL  l  BEHAVIORAL  SCIENCES 
5T01  EISENHOWER  AVENUE 
ALEXANDRIA,  VA  22333 

DR,  ROBERT  J.  SEIDEL 
US  ARMY  RESEARCH  INST. 

5D01  EISENHOWER  AVINUE 
3DD  N.  WASHINGTON  ST. 
ALEXANDRIA,  VA  22333 
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DR.  NORMAN  J.  KERR 
CH I £R  OF  NAVAL  TECH.  TRAINING 
NAVAL  AIR  STATION  MEMPHIS  (75) 
MILLINGTON,  TN  38054 


OR.  JAltES  McSRIDE 

NAVY  PERSONNEL  RSO  CENTER 

SAN  DIEGO ,  CA  92152 


Library.  Code  P201L 
Maw  Personnel  RSD  Center 
San  Diego,  CA  ?2152 


01 R.,  OFFICE  OF  MAM°OVER 
UTI LIZATiON 
HO..  MARINE  CORPS  (MPU) 

5C3 .  SLOG.  2709 
QUANT I  CO,  VA  22134 

OR.  A. L.  SLAFKOSKY 
SCIENTIFIC  AOVISOR  (CODE  RD-1) 
HQ,  U.S.  MARINE  CORPS 
WASHINGTON,  OC  20380 


MR.  JAMES  SAKER 
ARMY  RESEARCH  INSTITUTE 
5001  EISENHOWER  AVENUE 
ALEXANDRIA,  VA  22333 


PR.  MYRON  FISC.HL 

U.S.  ARMY  RESEARCH  INST. 

SOCIAL  S  BEHAVIORAL  SCIENCES 
5001  EISENHOWER  AVENUE 
ALEXANDRIA,  VA  22333 

COMMA.*! OER.  US  AFMY  RES.  INST. 

FOP.  BEHAVIORAL  S  SOCIAL  SCI. 
A7T11:  PERI -3 R( DP..  J.  OPASANU) 
?001  EISENHOWER  AVENUE 
ALEXANDRIA,  VA  22333 

DP.  ROBERT  5ASM0R 
L'S  ARMY  RES.  ' f«ST .  FDP.  THE 
BEHAVIORAL  &  SDCIAL  SCIENCES 
5001  EISENHOWER  AVI  VUE 
ALEXANDRIA,  VA  22333 

OR.  HILDA  WING 
ARMY  RESEARCH  INST. 

1331  t!tiW3Wi&  AVInue 
ALEXANDRIA,  VA  22333 


OR.  JIM  MOL LAN 
CODC  14 

NAVY  PERSONNEL  ROD  CENTER 
SAN  01  EGO,  CA  92152 

DR.  LEONARD  KROF.KER 
NAVY  PERSONNEL  RSO  CENTER 
SAN  1)1  EGO  .  CA  A3 152 


DR.  WILLIAM  MONTAGUE 

NPRDC  CODE  13 

SAN  DIEGO,  CA  92152 


TECHNICAL  DIRECTOR 
NAVY  PERSONNEL  RSO  CENTER 
SAN'  DIECO,  CA  92152 


Headquarters ,  US  Marine  Corps 
Code  HP  I -20 
Wasnington,  OC  20380 


MAJOR  FRANK  Y OH ANN AN .  USMC 
HEAOQUARTERS .  MARINE  CORPS 
(CODE  MP I -20) 

WASHINGTON,  DC  20330 


DR.  KENT  EATON 
ARMY  RESEARCH  INSTITUTE 
503!  EISENHOWER  ELVO. 
ALEXANDRIA,  VA  22333 


DR.  MILTON  S.  KATZ 
TRAINING  TECHNICAL  AREA 
U.S.  ARMY  RESEAP.CH  INST. 

5001  EISENHCMER  AVENUE 
ALEXANDRIA,  VA  22333 

JOSEPH  PSOTKA.  PH  .  D 
ATTN.  PERI-ID 
ARMY  RESEARCH  INSTITUTE 
5001  EISENHOWER  AVENUE 
ALEXANDRIA.  VA  22333 

DR.  JOYCE  SHIELDS 
ARMY  RESEARCH  INS".  “OR  THE 
BEHAVIORAL  S  SOCIAL  SCIENCES 
5001  EISENHOWER  AVENUE 
ALEXANDRIA,  V A  22333 

OR.  ROBERT  WISHER 
ARMY  RESEARCH  INST. 

5001  EISENHOWER  AVE . 
ALEXANDRIA,  VA  22333 


